Gradient-based learning of higher-order features

We describe an auto-encoder with multiplicative connections whose hidden variables encode products of pixel intensities. The model allows for efficient learning of image transformations and of higher-order structure within an image-patch. Modelling higher-order structure can be an effective way to improve recognition performance, as shown recently with a similar, probabilistic model of image correlations. In contrast to probabilistic models, auto-encoders can greatly simplify learning and inference, and they allow us to construct multi-layer processing hierarchies containing various pre-processing modules, that can be trained with simple back-propagation. We consider the task of learning higher-order image features, such as pair-wise products between pixel intensities. Unsupervised learning of pairwise products across images was considered by [6] as a means to encode image transformations. It has been extended and applied to various tasks, such as human action recognition [10], transparent motion estimation [7], foveal glimpse combination [5] and others. The model has also been extended to encode within-image pixel products, and was shown to yield state-of-the-art results in various object recognition tasks [8], [9]. An application of that model to language recognition has furthermore been shown to yield the best known result on the well-known TIMIT benchmark [1]. The models are based on hidden variables that are connected to observables (typically two images) in a three-way energy function, which, by exponentiating and normalizing, is turned into a probability distribution [6]. Learning amounts to approximate maximum likelihood, using a variation of contrastive divergence learning [3]. Learning is complicated somewhat by the fact that the basic model is defined using binary observations. Real-valued images are treated specially, and are known to require careful learning-rate control and normalizations to prevent “blow-up”. [9], [8] alternatively suggest using a variant of Hybrid Monte Carlo and various constraints on the model parameters to address these issues. In this work we describe an approach to learning higher-order features using simple gradient-based optimization. Our model is an auto-encoder modeling an image patch y. Connections between observable and hidden variables are multiplied by pixels in another image patch, x, forcing the hidden variables to encode pixel correlations rather than pixel means. The model is a type of higher-order neural network [2] applied to modeling feature correlations. Training amounts to simple back-propagation. This makes it straightforward to incorporate various pre-processing modules, such as low-dimensional projections to reduce computational complexity. In contrast to [7], there is no need for tensor factorization, or even for the manual calculation of gradients, for this purpose. Binary, real-valued and other types of data are dealt with in the usual way by deploying the appropriate activation/cost-functions in the final layer of the network. One can add standard hidden variables encoding first-order structure to the model, if desired, in which case the overall model may be called a “mean-covariance-encoder”. Figure 1 shows an illustration of a particular instance of the model, which can be viewed as non-probabilistic analog of the model described in [7]. The network takes two data-cases, x and y, as input, and projects these onto features (left-most vectors of triangles in the figure). These features are multiplied to provide the activations for hidden units. Reconstructions for y are computed similarly, by combining, multiplicatively, the vector of covariancevariables h with the (known) vector x. This defines a “conditional” model of y as a function x, similar to [6]. The